L15: Dictionary Methods

Theorizing and exploratory data analysis play a role in conceptualization

• theorizing helps with which aspects of a problem are likely to be important
• exploratory data analysis helps reveal previously hidden dimensions of a text

Our concept also needs to be reliable

• If measure our concept again, will we get the same answer?

Quntitative text analyses tend to be do well on the second dimension, but would need more work to demonstrate that concepts are also valid.

• Qualitative Aspect: Involves identifying concepts and creating categories or keys, associating them with specific textual features.

• Contextual Interpretation: Building dictionaries requires qualitative judgment and understanding of context.

• Quantitative Aspect: An algorithm is applied to large datasets, generating statistical summaries of text data.

• Reliability: Once constructed, dictionaries offer high reliability as analysis involves no further human interpretation.

Components of a Dictionary:

• Key: Represents a concept or category.
• Values: Terms or patterns associated with the key, functioning as equivalent instances of the concept.

Thus, the proportion of words in document \(i\) can be defined as:

\[t_{i}=\frac{\sum^{M}_{m=1} W_{im}}{N_i}\] where:

\(W_{im}\) - number of times word \(m\) appears in text \(i\) and 0 otherwise
\(N_{i}\) - number of words within a document

We need to divide by the number of words as we do not want texts to mechanically be assigned higher scores.

For example, stupid is more aggressive than accuse.

Not using weights means that all the words have the same weight.

However, sometimes we may want to use weights to express that some words are more refelctive of specific concepts.

1. Linquistic Inquiry and Word Count
   

• Different word categories reflecting psychological states, emotions, thinking styles, and social concerns

2. Lexicoder Sentiment Dictionary
   

• Negative and Positive Sentiment words designed for the automated coding of sentiment in news coverage, legislative speech and other text

3. Moral Foundations Dictionary
   

• Multiple-category dictionary of moral terms

4. Loughran-McDonald Sentiment Dictionary

• category sentiment dictionary, especially developed for financial analyses

5. Martindale’s Regressive Imagery Dictionary

• multiple category dictionary designed to measure primordial vs. conceptual thinking.

1. Words can have multiple meanings

• Loughran and McDonald classify sentiment for a corpus of 50,115 firm-year 10-K filings from 1994–2008
• Three-fourths of their negative words are not negative in a financial context: e.g. tax, cost, liability, foreign, vice, etc.

2. Dictionaries can lack important words in some contexts

• For example, there might be some negative financial words: felony, litigation, restated, misstatement

This is the question asked by Hargrave and Blumenau, 2022

We have the following columns within the dataframe:

Python

print(aggression_texts.columns)

Index(['person_id', 'name', 'constituency', 'entered_house', 'left_house',
       'date_of_birth', 'age_years', 'gender', 'house_start', 'days_in_house',
       'party', 'party_short', 'year', 'parliamentary_term', 'session',
       'pct_con', 'pct_lab', 'pct_ld', 'pct_other', 'margin', 'body',
       'question_time', 'debate_type', 'aggressive_word_count',
       'body_word_count', 'aggression_rating'],
      dtype='object')

And then examine the first 50 aggression words:

Python

# Convert aggression words to a set for faster lookup
aggression_words_set = set(aggression_words[0])
print(list(aggression_words_set)[:50])

['inadequacy', 'stupid ', 'prejudice ', 'falsehood', 'debase', 'hypocrisy', 'reprehensible', 'wrong ', 'coward', 'sneaky ', 'scapegoats', 'neglected', 'fool', 'cruel', 'blunder ', 'embarrasment ', 'backchat', 'incapable ', 'assault ', 'absurd ', 'mess', 'idiotic ', 'accusing ', 'groan', 'shameful', 'ludicrous', 'confrontational', 'outraged ', 'archaic ', 'deplorable', 'criticise ', 'negligent', 'silliness ', 'misleading', 'assaulting ', 'annoyed ', 'nasty', 'phony ', 'slander ', 'tactic ', 'aggressive ', 'failures ', 'betrayal', 'spite ', 'bitterly ', 'scandal', 'distasteful ', 'trickery ', 'crass', 'hypocracy']

The main concern here is whether texts are flagged for other reasons that have less to do with aggression.

There may be different types of validation depending on the research context.

• For example, British debates are much more aggressive during Question Time.

So, should we expect more agression during specific times?

However, even human judgment could be subject to many biases including:

• misinterpretation
• subjective idiosyncratic conceptualizations of agression
• lack of coder training

Asking human subjects to code speeches can also be expensive.

The aggression_texts data.frame includes a variable, aggression_rating.

This is the variable that that contains the ChatGPT rating.

ChatGPT was asked to rate aggression with 1 or 0.

There are 1,198 speeches that were categorized as aggressive by both the dictionary method and ChatGPT.

• these could be considered as true positives

We can think of this calculation as a measure of accuracy.

This measures the proportion of correctly classified cases (both positives and negatives) out of the total cases.

\[Accuracy = \frac{\textrm{True Positives}+ \textrm{True Negatives}}{\textrm{Total Observations}}\]

Applying it would entail a few steps:

1. Sum the counts for each class label in ChatGPT:

• Class 0: 17437 (false_false) + 608 (true_false) = 18045

• Class 1: 9400 (false_true) + 1198 (true_true) = 10598

• Class 0 is more frequent

2. The Naïve Guess Prediction: predicting 0 for all instances

Accuracy = (Number of correct predictions) / (Total number of instances)

• Correct predictions (all predicted as Class 0): True negatives (actual Class 0): 18045

• Total instances: 18045 (Class 0) + 10598 (Class 1) = 28643

• Naïve Guess Accuracy = 18045 / 28643 ≈ 0.63 (63%)

Our model’s accuracy (65%) > (63%).

The models captures some patterns beyond the most frequent class, but the improvement is modest.

• Measures the proportion of actual aggressive texts correctly identified.
• Formula: \(\text{Sensitivity} = \frac{\text{True Positives}}{\text{True Positives + False Negatives}}\)

Calculation: true_true/(true_true+true_false)

Sensitivity = 1198/(1198 + 608) ≈ 66.33%

• Interpretation: The ChatGPT model identifies about 66% of aggressive texts marked by the dictionary.

• Measures the proportion of non-aggressive texts correctly identified.
• Formula: \(\text{Specificity} = \frac{\text{True Negatives}}{\text{True Negatives + False Positives}}\)

Calculation: false_false / (false_false + false_true)

Specificity = 17437/(17437 + 9400) ≈ 64.97%

• Interpretation: The model correctly identifies approximately 64.97% of non-aggressive texts.

\[ \text{Accuracy} = \frac{\text{True Positives + True Negatives}}{\text{Total Observations}} \]

• Tells us how often the model makes the correct prediction overall.
• Limitation: Can be misleading if one class is much more common (class imbalance).

\[ \text{Sensitivity} = \frac{\text{True Positives}}{\text{True Positives + False Negatives}} \]

• High sensitivity means fewer aggressive texts are missed.

3. Specificity: Measures the model’s ability to detect non-aggressive texts.

\[ \text{Specificity} = \frac{\text{True Negatives}}{\text{True Negatives + False Positives}} \]

• High specificity means fewer non-aggressive texts are incorrectly marked as aggressive.

• Helps us see if the model performs better than a basic guess.
• Provides a minimum benchmark to assess model improvement.

• Judicial Decisions:
  • Often, we may prefer false negatives over false positives.
  • Question: Would you rather put an innocent person in jail or let a guilty one go free?

• COVID-19 Testing:
  • Here, false positives might be more acceptable than false negatives.
  • Question: Would you rather isolate unnecessarily or risk unknowingly spreading the virus?

• Accuracy: Proportion of correctly classified texts.
  • In your case: (true_true + false_false) / (true_true + false_false + true_false + false_true)
  • Calculation: (1198 + 17437) / (28643) ≈ 65.06%

• Sensitivity (Recall): Ability to correctly identify aggressive texts.
  • In your case: (true_true) / (true_true + true_false)
  • Calculation: 1198 / (1198 + 608) ≈ 66.33%

• Specificity: Ability to correctly identify non-aggressive texts.
  • In your case: (false_false) / (false_false + false_true)
  • Calculation: 17437 / (17437 + 9400) ≈ 64.97%

• In our case
  • Our aggression dictionary has a moderate success in detecting aggressive texts but shows limitations in both precision and recall.
  • It frequently misses a substantial portion of aggressive texts, resulting in moderate recall.
  • It sometimes misclassifies non-aggressive texts as aggressive

For judicial decisions, maybe we’d prefer false negatives than false positives - Would you rather put an innocent person in jail or let a guilty one go free?

For COVID tests, we might be more happy to accept false positives that false negatives - Would you rather isolate for no reason or spread the virus unknowingly?

• Start with a pre-existing dictionary of aggressive terms.
• Leverage word embeddings to:

• Expand the dictionary with terms relevant to parliamentary language.
• Upweight words frequently used in similar contexts within parliamentary speech.
• Downweight words less commonly used in similar contexts.

• Use these adjusted word lists to score speeches effectively.

Evaluating Classifier Performance: Beyond overall accuracy, it’s essential to consider within-class metrics like specificity and sensitivity for a nuanced assessment.

Benefits of Dictionary Methods: Fast and straightforward with many ready-to-use implementations, making them easy to apply.

Context Sensitivity: The validity of dictionaries depends on the context in which they were created and applied, which can limit their generalizability.

See for example this article in Social Science Computer Review

• Deep Contextual Understanding: LLMs grasp complex language patterns and nuances.
• Minimal Feature Engineering: Automatically learn relevant features from raw text.
• Scalability: Efficiently handle large volumes of text.